Remote set-up and calibration of an interactive system

ABSTRACT

Techniques for remotely calibrating an interactive system are provided. In one aspect, a three-dimensional model of the target environment is obtained, image and parameter data of the target environment are obtained, the image and parameter data are mapped onto the three-dimensional model, and calibration data of the target environment is developed based on the mapping. In another aspect, image and parameter data of the target environment is obtained, the image and parameter data is configured for transmission to a remote environment, calibration data is obtained from the remote environment, and display and sensor operation of the interactive system are updated based on the calibration data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.60/644,395, filed Jan. 15, 2005, incorporated by reference herein.

FIELD OF THE INVENTION

The present invention generally relates to interactive systems, and,more particularly, relates to set-up and calibration of such systems.

BACKGROUND OF THE INVENTION

The concept of ubiquitous computing is growing in popularity. A numberof technologies have been developed in connection with the ubiquitouscomputing field. For example, U.S. Pat. No. 6,431,711 to Pinhanezdiscloses a multiple-surface display projector with interactive inputcapability. In the Pinhanez patent, techniques are disclosed wherein animage is projected onto a surface in a room and is distorted beforeprojection so that a projected version of the image will not bedistorted. The surface can be planar or non-planar. The projected imagecan be displayed at multiple locations along a surface or multiplesurfaces. Thus, the projected image can move from one location on asurface to another location on that surface or another surface, whilethe projected image remains undistorted through the move.

Techniques using the Pinhanez invention allow interaction between peopleand a projector. Interactive input, such as from mice, can be used withversions of the Pinhanez invention. Versions of the Pinhanez inventioncan determine if an object is near an interactive item, such as ahyperlink, on the projected image. In such case, the interactive itemcan be activated. Thus, a person can interact with a projected image.

While it provides a substantial advance in the prior art, the Pinhanez'711 patent does not disclose techniques suitable for remote set-up,calibration and/or test of an interactive system. Local access to acomputer controlling the system is required, as well as the ability tophysically view various surfaces being calibrated for display.

SUMMARY OF THE INVENTION

Principles of the present invention provide techniques for remotelycalibrating, from a remote environment, an interactive system associatedwith a target environment. For example, one exemplary method can includesteps of obtaining a three-dimensional model of the target environment,obtaining image and parameter data of the target environment, mappingthe image and parameter data onto the three-dimensional model, anddeveloping calibration data of the target environment based on themapping. The three-dimensional model could be complete or could be apartial model including, for example, one or more of position, size, andorientation of essential display surfaces in the environment. Even asingle two-dimensional planar surface could be modeled, but the modelwould still be three-dimensional because it would typically includespatial orientation data for the surface.

In another aspect, an exemplary method could include steps of obtainingimage and parameter data of a target environment, configuring the imageand parameter data for transmission to a remote environment, obtainingcalibration data from the remote environment, and updating display andsensor operation of the interactive system based on the calibrationdata.

Principles of the present invention may be implemented, for example,using one or more computer systems, and may be embodied, for example, incomputer program products in the form of computer usable media and thelike.

These and other objects, features and advantages of the presentinvention will become apparent from the following detailed descriptionof illustrative embodiments thereof, which is to be read in connectionwith the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an exemplary embodiment of an apparatus for interactingwith a subject in an environment, according to one aspect of the presentinvention;

FIG. 2 depicts exemplary method steps for interacting with a subject inan environment according to an aspect of the present invention;

FIG. 3 depicts more detailed method steps for interacting with a subjectin an environment according to an aspect of the present invention;

FIG. 4 depicts a specific form of exemplary apparatus for interactingwith a subject in an environment according to another aspect of thepresent invention;

FIG. 5 depicts specific method steps for interacting with a subject inan environment according to an aspect of the present invention;

FIG. 6 depicts exemplary method steps that could be performed, forexample, by an image processor, in one or more exemplary embodiments ofthe present invention;

FIG. 7 depicts an exemplary system for remote calibration and/or setupof an apparatus in accordance with one or more aspects of the presentinvention;

FIG. 8 depicts exemplary method steps for remotely calibrating aninteractive system associated with a target environment, from a remoteenvironment, in accordance with still a further aspect of the presentinvention;

FIG. 9 depicts an exemplary apparatus for collocating a processing unitand a visual unit according to still a further aspect of the presentinvention;

FIG. 10 depicts the apparatus of FIG. 9 without the visual unit inplace;

FIG. 11 depicts the apparatus of FIGS. 9 and 10 with the processing unitvisible;

FIG. 12 depicts the apparatus of FIGS. 9-11 in an alternative mountingconfiguration;

FIG. 13 depicts an interior portion of a processing unit receivingportion of the apparatus of FIGS. 9-12;

FIG. 14 depicts part of a visual unit mounting portion of the apparatusof FIGS. 9-13;

FIG. 15 shows another view of the portion of FIG. 14;

FIG. 16 is an exploded view of the portion of FIGS. 14 and 15 inrelation to an adjacent portion of the apparatus;

FIG. 17 depicts a visual unit receiving portion of the apparatus ofFIGS. 9-16;

FIG. 18 is an exploded view of the portion of FIG. 17 with a visual unitcontained therein;

FIG. 19 shows details of an exemplary tilt bracket useful with theapparatus of FIGS. 9-18; and

FIG. 20 depicts an exemplary computer system that can implement one ormore method steps and/or elements of one or more aspects or embodimentsof the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

FIG. 1 depicts a block diagram of an exemplary apparatus 100 forinteracting with a subject 102 in an environment 104, according to oneexemplary embodiment of the present invention. Apparatus 100 can includeone or more display devices 106 configured to display images such as 108on a number of surfaces such as 110 in the environment 104. The displaydevices are designated as D₁, D₂, . . . D_(r).

Apparatus 100 can also include one or more sensing devices 112 that areconfigured to sense interactions between the subject 102 and theenvironment 104. In one or more aspects of the invention, theinteractions that are sensed can include interactions that are dispersedthrough the environment 104 and are remote from the images 108 displayedby the display device. The sensing devices are designated as S₁, S₂, . .. S_(n).

Apparatus 100 can further include one or more display controllers suchas display controller 114. Controller 114 can be coupled to the displaydevice or devices 106 and the sensing device or devices 112, and can beconfigured to modify image display by the display device or devices 106in response to one or more of the interactions sensed by the sensingdevice or devices 112. Such modification of the image display caninclude modification of one or more of image location, imageorientation, and image content. It should be noted that displaycontroller 114 can be implemented as a single unit 114 as shown, or maybe realized as a collection of individual display controllers for eachdisplay device 106, or as some combination of the foregoing.

One or more of the images 108 can include an interactive item or items116 (analogous to hyperlinks), and the display controller 114 can beconfigured to activate an interactive item such as item 116 when anobject is close to the interactive item. Such an object could include,by way of example and not limitation, one or more of the body of subject102, a hand of subject 102, a finger of subject 102, an obstructionmanipulated by subject 102, or even an illuminated region. In theexample depicted in FIG. 1, the object is a laser-pointer-illuminatedregion 118 that is illuminated by action of the subject 102, namely, byaiming a laser pointer 120.

In view of the foregoing discussion, it will be appreciated that some ofthe exemplary types of interactions that can be sensed by the sensingdevices 112 include proximity of an object to one of the surfaces suchas surfaces 110, or the performance of a predefined motion by thesubject 102 proximate to one of the surfaces 110. Such predefined motioncould include, for example, touching and/or pointing.

The sensing devices 112 can sense and/or monitor various parts of theenvironment 104. The sensing devices 112 can be of different types. Forexample, some of the sensing devices can be optical cameras, otherscould be radio frequency identification tags, others could bemicrophones, and still others could be pressure sensors. Each of thesensing devices 112 can be configured to monitor different parts of theenvironment 104. By way of example and not limitation, in an environmentsuch as a retail store, different cameras could be viewing differentaisles in the store. Radio frequency identification tags may be presenton products on the shelves in the store. Pressure sensors may be presentin certain areas, such as the entrance to an aisle. Microphones might bepresent at selected locations on walls, for example, with an “ask forhelp here” sign located next to them. Again, the foregoing is intendedas an illustrative example, not intended to be limiting, regarding themanner in which different types of sensing devices can sense and monitordifferent activities taking place in the environment 104. As noted, inone or more exemplary embodiments of the invention, one or more of thesensing devices can be configured to sense interactions (between thesubject and the environment) that are dispersed through the environment104 and remote from the images 108 displayed by one or more of thedisplay devices 106. Again by way of example and not limitation, suchdispersed interactions could be sensed by the aforementionedmicrophones, pressure sensors, or radio frequency identification tags,or by optical cameras focused on areas other than the images 108.

It should be noted that elements such as the one or more display devices106, one or more sensing devices 112, and one or more sensor processors122, to be discussed more fully below, are shown in block diagram formin FIG. 1 adjacent display controller 114. This is for purposes ofillustrative convenience. It is to be appreciated that one or more ofsuch display devices 106, sensing devices 112, and even, if desired,sensor processors 122 would typically be located in or adjacentenvironment 104. Two different display devices designated as 106A and106B are shown within environment 104 for illustrative purposes, as isone sensing device designated as 112A. Environment 104 could, of course,contain a variety of display devices 106 and sensing devices 112 in anumber of different locations. By way of example, a retail store mighthave large plasma displays such as 106A on some walls. Smaller liquidcrystal displays could also be present in one or more locations on theshelves. Some of these displays could be on railings along the shelves,and provision could be made for relocation to different portions of theshelf through electronic control.

Another type of display could be a video projector, such as shown at106B, which projects on a wall, or an area of the floor, etc., such ason surfaces 110. One or more of the projectors, such as 106B, could bemovable, or the images therefrom could be movable using a redirectiondevice such as a pan/tilt mirror as in the aforementioned U.S. Pat. No.6,431,711 Pinhanez. Another possible type of redirection device is amotorized pan/tilt unit as available from Directed Perception Inc. underitem number PTU-D46-17 or PTU-D46-70. Accordingly, appropriateredirection can be employed with such movable projectors, or projectorshaving redirection devices, such that images can be displayed ondifferent surfaces within the environment 104.

It will be appreciated that one or more exemplary embodiments of thepresent invention enable sensing of the subject 102 (broadly understoodto also include objects adjacent to, associated with, or under controlof the subject) and interactions of the subject with the environment104. Further, the location, orientation or content of one or more imagescan be changed in response to the sensed interactions, using one or moreof the display devices 106. As noted above, apparatus 100 can furtherinclude one or more sensor processors 122. Such processors 122 can becoupled to the sensing device or devices 112 and to the displaycontroller 114, and can be configured to receive a data stream from thesensing device or devices 112. One or more sensor processors 122 can beemployed; in the exemplary embodiment depicted in FIG. 1, each sensingdevice 112 has a sensor processor 122 associated therewith. Each sensorprocessor 122 can receive a data stream from the associated sensingdevice 112, and can make an appropriate output. For example, each sensorprocessor 122 can output a set of objects O_(i) and a set of interactionparameters K_(i). The object parameters O_(i) can be numbered O₁, O₂, .. . O_(n) (the subscripts can be omitted in further discussion forconvenience). Similarly the interaction parameters K_(i) can be numberedK₁, K₂, . . . K_(n) (again the subscripts can be omitted in furtherdiscussion for convenience).

The set of object parameters could include, e.g., the number of objects,the locations of each object, and the label of each object. In a retailenvironment the objects could be human beings, specific products,shopping carts, and the like. The set of interaction parameters couldinclude, for example, the number of interactions detected, the locationof each interaction, and the type of each interaction. Again by way ofexample and not limitation, in a retail environment, examples ofinteractions include a person walking into an aisle, a shopper pickingup an object, a user touching a display, or a shopping cart traversingthe aisle. As noted, when reference is made to interacting with asubject in an environment, this should be broadly understood to includeinteractions with any of the foregoing objects present in theenvironment that are under the control of or otherwise associated withor adjacent the subject (for example, by virtue of being accessible tothe subject in the environment).

The outputs of the sensor processors 122 can be received by displaycontroller 114, along with a number of data streams V₁, V₂, . . . V_(n).At each instance of time “t” the display controller(s) can also have oneor more inputs designated generally as 124. Such inputs 124 cancorrespond to the status of the display devices 106 at the previousinstance of time “t−1.” The parameters 124 could include, by way ofexample and not limitation, an image I_(j) displayed by the displaydevice D_(j) and a set of parameters P_(j) that define one or more ofthe position, size and orientation of the pertinent displays. Based onthe outputs of the sensor processors 122 and the display deviceparameters 124 at the previous instance of time, the display device(s)can determine new display device parameters I¹ _(t), I² _(t), . . .I^(r) _(t) and/or P¹ _(t), P² _(t), . . . P^(r) _(t). Such parametersthus could include the image I to be currently displayed by each displaydevice 106, and the set of position parameters P for that device. Thus,the display controller 114 can be configured to determine one or more ofcontent, location and orientation of the images displayed in theenvironment 104, based on sensed objects and/or interactions in theenvironment. For example, in a retail environment, the entry of a personinto an aisle may result in a welcome message displayed on the floor ofthe aisle in an orientation suitable for viewing by that person. Inanother example, when a person picks up a product from a shelf, thesystem may use a set of consecutive displays, both static displays suchas 106A, and a movable projection display such as 106B, to guide theperson to a related product that is on sale.

It will be appreciated that the exemplary apparatus 100 depicted in FIG.1 can be configured in a number of different manners. One or moredisplay devices 106, sensing devices 112, and/or sensor processors 122can be employed in different locations and in different combinations.Thus, by way of example and not limitation, in addition to, oralternatively to, the aforementioned configuration of a sensing device112A to sense interactions dispersed through the environment 104 andremote from the images 108 displayed by the display device 106B, inanother aspect or configuration, the apparatus 100 can be configuredwith a projector plus an additional physical display. Thus, apparatus100 can be configured with one or more projectors 106B configured toproject images 108 on a number of surfaces 110 in the environment 104.Further, at least one physical display such as 106A can be located inthe environment 104 and can be configured to display images to thesubject. At least one sensing device 112 can be configured to senseinteractions between the subject 102 and the environment 104, but allconfigurations do not necessarily require that this sensing device 112be configured to sense interactions that are remote from the images(that is, in some configurations or embodiments, the sensing device 112might sense just interactions adjacent to the images).

The at least one display controller 114 can be coupled to the projector,physical display, and sensing device and can be configured to modifyimage display of the projector and the physical display 106A, responsiveto one or more interactions sensed by the sensing device. The displaycontroller 114 can be adapted to distort an undistorted image into adistorted image for projection by the projector 106B. The apparatus canfurther include a projector mounting, of a type to be described below,to which the projector 106B is mounted. The projector mounting could beconfigured to perform one or more of rotating and translating theprojector 106B to project the distorted image on one or more of theplurality of surfaces 110. The mounting could, if desired, be a combinedmounting wherein the aforementioned camera and projector 106B arecollocated for substantially coupled motion. The image could bedistorted such that when projected on one or more of surfaces 110, itappears in substantially undistorted form.

Display controller 114 could include a video adapter, and could beconfigured to set parameters of a correction surface in the videoadapter and to place the undistorted image as a texture on thecorrection surface. The display controller could also include, forexample, focus and/or zoom control, position control, and the like.Further, the display controller could be configured to set parameters ofa transformation matrix to distort the undistorted image into adistorted image. As discussed below, projector 106B could be configuredfor both rotational and translational movement with respect toenvironment 104. The aforementioned correction surface could haveparameters including three rotational coordinates, two translationalcoordinates, a lens parameter, and a scale parameter. One or more of thesensing devices 112 could be a camera configured to sense interactions.One or more of the sensor processors 122 could be image processorsconfigured to receive a video data stream from an associated camera, andto output a set of object parameters O and a set of interactionparameters K to the display controller 114. The object parameters andthe interaction parameters could be of the type described above. Asnoted, one of the sensors 112 can be a camera that is configured toreceive a camera image. Such image can be formed by a reflection fromone of the surfaces 110 of one of the images projected by the projector106B. The display controller 114 can be configured to communicate zoomand focus information to the projector 106B. The aforementioned imageprocessor or image processors can be configured, for example, to performmethod or processing steps to be discussed with respect to FIG. 6 below.

The exemplary apparatus of FIG. 1 can be configured in a number ofdifferent ways. Thus far, (i) an exemplary configuration wherein one ormore dispersed interactions may be sensed, and (ii) an exemplaryconfiguration including a projector plus an additional physical display,have been described. A number of other configurations are possible. Forexample, the sensing devices 112 can include at least one cameraconfigured to sense interactions between the subject 102 and one or moreof the images 108, and at least one additional sensor. In such aconfiguration, at least one display device 106 could be provided todisplay images 108 on a plurality of surfaces 110 in environment 104 asdiscussed above. The additional sensor can be configured to senseinteractions between the subject 102 and the environment 104 in(dispersed) regions remote from the images 108. Such sensor could be, byway of example and not limitation, a radio frequency identification tag,a pressure sensor, a microphone, and the like, or even a camera focusedon regions away from the images 108. An image processor, as discussedabove, could be included and could be coupled to the camera and thedisplay controller, and configured to process images from the camera soas to detect objects in the environment, and further to detectinteractions of the objects with the plurality of surfaces. The imageprocessor could again be configured to perform steps as will bediscussed with respect to FIG. 6 below.

Actions described above as being performed by various elements depictedin FIG. 1 (or in other figures described herein) can be performedaccording to one or more exemplary methods in accordance with one ormore aspects of the present invention (inventive methods are notnecessarily limited to being performed by exemplary apparatuses). By wayof example and not limitation, FIG. 2 shows a flow chart 200 ofexemplary method steps, which can be computer-implemented, forinteracting with a subject in an environment, according to an aspect ofthe present invention. The start of the method is indicated at block202. At block 204, a method step can include displaying images on aplurality of surfaces in the environment. Such images can includedisplay of more than one image at the same time, or could includedisplay of more than one image at sequential times, for example, asingle image at a time, but with changes over time. As shown at block206, interactions between the subject and the environment can be sensed.At least some of the interactions can be dispersed through theenvironment and remote from the images displayed in block 204. Thedisplay of the images can be modified responsive to one or more of theinteractions sensed by the sensing device in block 206. One possiblemanner in which such image display can be modified will be discussedbelow with regard to the remainder of FIG. 2.

In one or more exemplary embodiments of the present invention, thesurfaces on which the images are displayed can be divided into one ormore display zones. As shown at decision block 208, a determination canbe made whether any of the display zones are adjacent a given one of theinteractions that has been sensed. If it is determined that at least oneof the display zones is adjacent to the given one of the interactions,as indicated by the “Y” branch emanating from decision block 208, theadjacent display zone can be selected for display, as at step 218, of anupdated image that is modified in accordance with the given one of theinteractions.

The images can be displayed, for example, by display devices in thedisplay zones. A determination can be made, as shown at block 220,whether one or more of the display devices in an adjacent display zoneis movable. If such is the case, as shown at block 222, the givendisplay device can be moved to a new location (as required). Regardlessof whether or not the device is movable, the updated image can bedisplayed as at block 224. If none of the display zones is adjacent thegiven one of the interactions, as indicated at the “N” branch emanatingfrom decision block 208, a determination can be made, as at block 210,whether some other kind of notification to the subject is possible. Ifsuch alternative notification is possible, as indicated at the “Y”branch of decision block 210, the notification can be sent as indicatedat block 212. Such notification could be, for example, by means of aloudspeaker or a flashing light. If no other notification is possible,and/or if an appropriate display has been made, the given interactioncan be removed from a list of interactions, as shown at step 214. Asindicated at block 216, one can continue to process, or if one hadcompleted all desired processing, one could stop (the terminology “one”is intended to encompass performance of the steps by a computer).

FIG. 3 shows a flow chart 300 of exemplary detailed method steps forsensing interactions and responding through appropriate displays. Sensordata streams are processed in a processing step 302, preferably by asensor processor 122 of the kind discussed above, to detect the presenceand/or location of objects in an environment. A list of objects togetherwith processed sensor data are received by another processing step 304,wherein object interactions are detected. Each given interaction in aresulting list of interactions can be processed in step 306 in order todetermine available display zones near the location of the particularinteraction. A display zone can indicate an area in which an image canbe displayed via one or more display devices in the environment.Examples of display zones can include the display area of a staticdisplay, or an area of the floor or wall that could be displayed upon bya moveable or redirectable display. Accordingly, step 306 can employcurrently available display device parameters as indicated.

A decision block 308 determines whether any of the display zones iswithin an acceptable range, for example, in terms of parameters such aslocation, orientation and size, from the selected interaction. If suchis not the case, a further decision step 310 determines whether it isnecessary and/or possible to provide an alternative notification (asdiscussed above) in response to the sensed interaction; for example,such alternative notification could be a voice message or a beep. If thedecision in block 310 is affirmative, the appropriate notification canbe selected and sent via processing step 312. If, conversely, thedecision in 310 is negative, no display device need be activated, andthe current interaction can be removed from the list of interactions asin step 314. The updated list of interactions can then be processed instep 306, continuing in a looping fashion until all detectedinteractions are processed. Note that if the notification is sent inblock 312, the interaction can also be removed from the list, and thelooping can continue, as just described.

If the decision in step 308 is affirmative, and there are one or moredisplay zones available within an acceptable range for the sensedinteraction, at step 316, a display zone and display device can beselected. At step 318, an image that is to be displayed can be obtained.Such image may come, for example, from a separate application such as anadvertising application, or may be created within step 318. The contentof the image obtained in step 318 can then be modified in step 320,based on the sensed interaction. Further, in decision block 322, adetermination can be made whether the selected display device is staticor movable. If not moveable, one can proceed directly to block 324, andthe modified image can be sent to the selected display device.

If it is determined in step 322 that the selected display device ismovable, at block 326, a determination of the appropriate devicelocation and orientation parameters for the moveable devicecorresponding to the selected display zone can be made. In step 328, theselected display device can be moved to its new location andorientation, for example, by sending appropriate control signals. Atstep 330, appropriate parameters to distort the undistorted imageobtained in step 320 can be determined, such that when displayed by themoveable display device, a substantially undistorted image appears onthe selected surface and within the selected display zone. Techniquesfor distorting the image in a moveable projection system are known, forexample, from the aforementioned U.S. Pat. No. 6,431,711 to Pinhanez.Appropriate image distortion can be performed as indicated at step 332,and the properly distorted image can be sent to the movable displaydevice as in the aforementioned step 324 (distortion may not be neededfor displays other than projectors). The currently selected interactioncan then be removed from the list of interactions as at step 314, andthe updated list of interactions can be processed in step 306, againcontinuing with a loop processing all desired detected interactions.Accordingly, it will be appreciated that FIG. 3 depicts a flow chart 300of a detailed method for sensing object interactions and responding tothe interactions by providing, where appropriate, a substantiallyundistorted image upon a surface.

FIG. 4 shows a specific form of exemplary apparatus 400 for interactingwith a subject in an environment, according to an aspect of the presentinvention. Elements similar to those described above with respect toFIG. 1 have received the same reference character incremented by 300.Display device 406 is in the form of at least one video projectorconfigured to display images on a number of surfaces in the environment.A video projector mounting 426 is provided, coupled to the videoprojector 406, and configured to move the video projector in at leasttwo rotational degrees of freedom, for example, pan and tilt, asindicated by the arrows in the figure. One or more translational degreesof freedom can also be provided, as could, if desired, an additionalrotational degree of freedom. Sensing device 412 is in the form of atleast one video camera configured to sense interactions between thesubject and the environment.

Display controller 414 can be coupled to the video projector 406, videoprojector mounting 426, and video camera 412. Controller 414 can includea memory and at least one processor. The processor can be coupled to thememory, and can be operative to modify image display by the videoprojector 406 responsive to one or more of the interactions sensed bythe video camera 412. Processor 414 can also be operative to makedeterminations regarding display zones, as described above with respectto FIGS. 2 and 3, and below with respect to FIG. 5. The processor can befurther be operative to perform image distortion as described above, tooutput a distorted projection image to the video projector 406, and tocause; the video projector mounting 426 to move so as to in turn causethe video projector 406 to project the distorted projection image on adesired one of the surfaces within the environment. Further, theprocessor can be operative to communicate desired pan, tilt, and otherrotational or translational parameters to the video projector mounting426. The processor can be further configured so as to be operative toundistort an image, or a series of images, from the camera 412 to obtainan undistorted camera image or a series of undistorted camera images. Acomparison can be made between a camera image (preferably undistorted)and a projection image (also preferably undistorted) to determinewhether an object is present in the image from the camera. Alternativelyor additionally, a comparison can be made among successive camera images(preferably undistorted) to determine whether an object is present in agiven one of the images from the series of images from the camera.

In one preferred form of apparatus 400, a single video camera 412 and asingle video projector 406 are employed. The processing capability canbe provided, for example, via a computer system 448 of a type to bediscussed in connection with FIG. 20 below. The video projector 406could be, for example, an XJ-450 Digital Light Processing (“DLP”)projector from Casio, Inc. The video camera 412 could be, for example, adevice such as the LogiTech for Notebooks Pro from LogiTech, Inc. Thevideo projector mounting 426 could be, for example, a pan/tilt head suchas device PTU D46-17 from Directed Perception, Inc. The computer 448could be, for example, a notebook computer such as the IBM ThinkPad T42from IBM Corporation. The sensor or video processor 422 and the displaycontroller 414 could both be implemented by employing the aforementionedcomputer 448, for example, using appropriate software routines ormodules running on the computer.

Image processor 422 could process the sequence of images V coming fromthe camera 412, and could output a set of parameters including theobject parameters O and interactions K as described above. Interactionssensed by camera 412 could include, for example, presence of a person inan aisle, the act of picking up a product, a hand touching an imagedisplayed by the projector 406, and the like. The parameters and theimage sequences could be processed by the processor 414. Processor 414can also receive the parameters 424 including the image I displayed bythe projector 406, and the parameters P of the video projector mounting426, corresponding to the previous time instance t−1. The parameters Pcan include, for example, pan value, tilt value, pan and tilt speeds,current position and orientation, focus and zoom of the projector 406,as well as image distortion parameters. Based on the sensed interactionK, the controller 414 can determine a new image to be displayed by theprojector 406 at the current time instance t, as well as new parametersfor the video projector mounting 426. Accordingly, apparatus 400represents one possible form of a steerable interactive system.

FIG. 5 shows a flow chart 500 of one possible exemplary method forsensing interactions and responding thereto in accordance with an aspectof the present invention, and can be applied, for example, to theapparatus depicted in FIG. 4. Video from a camera, such as, for example,a sequence of images in time, is processed at steps 502, 504, preferablyby an image processor such as 414, to detect the presence and locationof objects in an environment. In step 502, image differencing isperformed by computing the difference between a current image seen bythe camera, and a previous image seen by the camera. Alternatively, theimage differencing could include computing the difference between theimage seen by the camera, and the image projected by the projector.Additional shape analysis can be performed in step 504, on foregroundregions which can be detected in step 502, so as to obtain a list ofobjects and appropriate processed image data. The list of objects andthe processed image data can be received in step 506, where motionanalysis can be performed based on the aforementioned list of objectsand processed image data, to obtain a list of interactions. Theinteractions in a list of interactions can be processed in step 508 todetermine available display zones near the location of the particularinteraction.

A display zone can indicate an area in which an image can be displayedvia the movable video projector. In step 508, the currently availabledisplay device parameters can be employed in the determination. Indecision block 510, a check can be made whether any of the display zonesis within an acceptable range, in terms of parameters such as location,orientation, and size, from the selected interaction. If such is not thecase, at additional decision block 512, a check can be made whetherother notification is necessary and/or possible in response to thesensed interaction. As noted above, such alternate notification couldbe, for example, a voice message or a beep. If the decision in block 512is affirmative, the appropriate notification can be selected and can besent as per step 514. In the event that the decision in block 512 isnegative, no display device needs to be activated, and the currentinteraction can be removed from a list of interactions as at step 516.The updated list of interactions can then be processed in step 508, thusforming a loop for processing of detected interactions. Note that if thenotification is sent in block 514, the interaction can also be removedfrom the list, and the looping can continue, as just described.

If the decision in block 510 is affirmative, and there are one or moredisplay zones available within an acceptable range for the sensedinteraction, at step 518, one can select a display zone from among thosewithin the acceptable range. In step 520, the image to be displayed canbe obtained. As noted above, such an image may come from a separateapplication such as an advertising application, or may be created withinstep 520. The content of the image obtained in step 520 can be modifiedin step 522, based on the sensed interaction.

In step 524, a determination can be made regarding appropriateredirection parameters corresponding to the selected display zone. Instep 526, the projector can then be moved to its new location andorientation by sending appropriate control signals to the videoprojector mounting, which can also be referred to as a redirectiondevice. In step 528, image distortion parameters can be estimated asneeded to distort the undistorted image obtained in step 522, such thatwhen displayed by the moveable video projector, a substantiallyundistorted image will appear on the selected surface within theselected display zone. Image distortion has been discussed elsewhereherein, and can be performed as indicated in step 530, with thedistorted image sent to the moveable display device in step 532. Thecurrently selected interaction can then be eliminated from the currentinteraction list in step 516, and the updated list processed in step508, thus continuing the loop for processing of the detectedinteraction(s). Thus, FIG. 5 can be viewed, in a sense, as a specialcase of FIG. 3, and can be thought of as depicting a method capable ofprojecting a substantially distortion-less image on any of a multiplenumber of surfaces, and for providing interaction with the substantiallydistortion-less image, or response to other interactions in theenvironment, according to one aspect of the present invention.

FIG. 6 depicts a flow chart 600 of exemplary method steps, in accordancewith an aspect of the present invention, which can be performed, forexample, by the above-mentioned image processor in connection with oneor more exemplary embodiments of the present invention. As indicated atstep 602, a distorted camera image can be undistorted to create anundistorted camera image. In step 604 a difference image can be found.In one aspect, the difference image can be between the (preferablyundistorted) camera image, and a (preferably undistorted) form of theimage projected by the projector. In another aspect, the differenceimage can be the difference between the (preferably undistorted) cameraimage and a previous (preferably undistorted) camera image. As depictedat block 606, the difference image can be thresholded and filtered.Noise can be removed, and the thresholding and filtering can yieldforeground regions that correspond to objects within a region of theenvironment viewed by the camera. The shape and movement of theforeground regions can be analyzed at block 608, to detect interactionsby the objects on the plurality of surfaces in the environment.

In one or more aspects or embodiments of apparatuses and methods inaccordance with the present invention, a computer or other components ofembodiments of the present invention may not be readily accessible for ahuman to control, drive, and/or set up or calibrate. Accordingly,techniques for remote set-up, calibration, and/or control of systems andmethods according to the present invention may be desirable.

FIG. 7 depicts an arrangement 700 useful, e.g., in remote control,set-up and/or calibration of systems according to one or more aspects ofthe present invention. By way of example and not limitation, remotecontrol of a system similar to that shown in FIG. 4 is depicted in FIG.7, and elements 706, 712, 714, 722, 724, 726, and 748 can function in amanner similar to those elements depicted in FIG. 4 that have the samereference character less 300. Accordingly, functioning of such elementswill not be described again.

The local computer 748 can be connected, through a network connection750 such as an Ethernet or wireless connection, to a remote computer752. Remote control software such as the PCAnywhere software fromSymantec, Inc. can be employed to enable replication and control of thedisplays and the interfaces on the local computer 748 at the remotecomputer 752. An appropriate user interface 754 can be provided on thelocal computer to enable user interaction with the system for set-up,viewing, control, and the like. Interface 754 can communicate with thedisplay controller 714 to achieve appropriate display and controlfunctions as required or desired for user interaction.

The replication of the entire desktop interface of the local computer748 on the remote computer 752 can enable access to the user interfaceby a remote user 756 through the replicated interface 758. Accordingly,remote user 756 can perform functions such as set-up and/or control ofthe local system depicted in the left-hand portion of FIG. 7. Thereplication is suggested by elements 760, 762, and it will beappreciated that elements 758, 760, 762 in the replicated portioncorrespond to elements 754, 722, 714 respectively in the local system.

During calibration of the local system, location, orientation, and sizeof one or more display zones in the environment may be defined by theuser. Accordingly, the user should be able to view the environment whileprojecting images and manipulating them through the user interface 758.Camera views of the local environment may not permit proper calibration,as the cameras may present distorted views of the environment due tocamera geometry. Thus, during calibration, a viewer might need to be inthe target (local) environment even if a remote computer were being usedto set up the system. Accordingly, it is desirable to be able toeffectively calibrate the system from a remote location.

FIG. 8 shows a flow chart 800 of exemplary method steps that can beemployed, for example, with the elements depicted in FIG. 7, to performsuch remote calibration. One or more method steps described with respectto FIG. 8 can be computer-implemented, as can other method stepsdescribed herein. The left-hand portion of the flow chart is labeled“Remote Environment” while the right-hand portion of the flow chart islabeled “Target Environment.” The portion labeled “Remote Environment”depicts method steps that can be performed by a remote computer beingused by a user in a remote environment. The portion labeled “TargetEnvironment” depicts steps that can be performed on the system computerin the target environment.

As shown in step 802, a three-dimensional model of the targetenvironment can be obtained. It should be noted that such athree-dimensional model could be a partial model; for example, it mightinclude only position, size, and orientation of one or more essentialdisplay surfaces in the environment. Indeed, the three-dimensional modelmight include a model of only a single two-dimensional planar surface,however, it would still be a three dimensional model because spatialorientation data of the two-dimensional planar surface would typicallyalso be included. Stated in another way, the three-dimensional modelmight not include every book on a given shelf within the environment.Image and parameter data of the target environment can be obtained assuggested by the dotted line from block 828 under “Target Environment”to block 804 under “Remote Environment.” The image and parameter datacan be mapped onto the three dimensional model of the targetenvironment, for example, in simulation step 804. As shown within block805, a number of steps can be employed in the process, culminating inthe development of calibration data of the target environment based onthe mapping.

The aforementioned three-dimensional model of the target environment canbe maintained on the computer in the remote environment. In one or moreembodiments, the model can be built beforehand based on measurementsmade in the target environment either manually or through automated orsemi-automated techniques. In the simulation step 804, the remotecomputer can simulate a view of the target environment based on theaforementioned three dimensional model as well as the aforementionedprojector and camera images received from block 828 under “TargetEnvironment.” method. In the simulation, the current projector andcamera images can be mapped onto the surfaces of the three-dimensionalmodel of the target environment. In step 806, the simulated environmentcan be displayed to a user through an interactive interface. The usercan be permitted to view, update, and/or manipulate current positions ofthe camera and projector, or other sensing and display devices, and tocorrect one or more of location, orientation, size, and distortion ofimages through the simulated environment. As the user manipulates thethree-dimensional model, the user is able to view the environmentsubstantially without distortions that might be present if mere cameraviews of the target environment were employed.

In decision block 808, a determination can be made whether the userneeds to further update the environment. If such is the case, suchupdates can be performed in block 810, and updated parameters can besent to step 804, resulting in a new simulation and display to the userin block 806. If no further update is required in block 808, newparameters for the redirection device, focus and zoom parameters of theprojector and/or camera, image distortion parameters for the projectedimage in the target environment, and the like can be computed anew instep 812, based on the last value selected by the user in the simulationenvironment. In step 814, such new parameters can be transmitted back tothe system computer in the target environment.

In view of the foregoing discussion, it will be appreciated that thedevelopment of calibration data of the target environment based onmapping can include one or more of updating a portion of the image andparameter data to obtain updated image and parameter data, mapping theupdated image and parameter data onto the three-dimensional model, andconfiguring the updated image and parameter data for transmission to theinteractive system associated with the target environment. Suchtransmitted data can be employed as calibration data. The updating andmapping steps can be repeated until it is determined, during themapping, that the updated image and parameter data is satisfactory. Theresults of the repeated steps can then be configured for transmission atblock 814. The aforementioned image and parameter data can include oneor more of focus data, zoom data, pan and tilt data, image data from acamera in the remote environment, and image data from a projector in theremote environment. The image and parameter data could also includedisplacement data, where the projector or other display device in thetarget environment is capable of translational as well as rotationalmotion. As noted, the three-dimensional model of the target environmentcould be built beforehand, and accordingly, one or more embodiments ofthe method can include an additional step of building the aforementionedthree-dimensional model of the target environment.

In view of the foregoing discussion, it will be appreciated that in oneor more other aspects of the invention, the developing step couldinclude displaying results of the mapping step to a user, obtaining userinput defining updates to at least a portion of the image and parameterdata, so as to obtain updated image and parameter data, repeating thedisplaying step and the step of obtaining user input until the userdetermines that the updated image and parameter data is satisfactory,and configuring the updated image and parameter data for transmission tothe interactive system associated with the target environment, again,for use as calibration data.

It should be noted that one or more of the aforementioned steps might beperformed by an external system, apparatus, process, or method. Forexample, a software module implementing techniques of the presentinvention might simply format data for display to a user, and/or formatparameters for transmission to the target computer, and such displayand/or transmission, for example, could be performed by one or moresystems, apparatuses, processes, or methods external to routinesimplementing the present invention.

Attention should now be given to the right-hand portion of FIG. 8, fordiscussion of an exemplary method for remotely calibrating, from aremote environment, an interactive system associated with a targetenvironment, in accordance with another aspect of the present invention.In step 816, current parameters of a redirection device can be read inthe target environment. Such parameters, together with appropriateprecalibration data, can be employed in step 818 to estimate projectorand/or camera locations and/or orientations. Current focus and zoomparameters of the projector and camera can be read in step 820. Thecurrent image sent to the projector for display can then be read in step822, while in step 824, the current image seen by the camera can beread. It will be appreciated that the aforementioned steps essentiallyconstitute one possible way to effectuate the obtaining of image andparameter data from the target environment, as indicated in dotted block826. Such data can be configured for transmission to the remoteenvironment, and as shown in block 828, can be transmitted to the remotecomputer.

New parameters, such as the aforementioned calibration data, can be readfrom the remote environment in step 830. In step 832, new distortionparameters can by applied to the image to be displayed. In step 834, theupdated image can be sent to the projector for display. In 836, thefocus and zoom parameters of the projector and camera can be adjustedand/or updated. In step 838, the redirection device can be driven to anew location based on the received parameters. Accordingly, it will beappreciated that one or more of the previous steps essentially setforth, as indicated by dotted block 840, a possible way or way(s) toperform a step of updating display and sensor operation of theinteractive system based on the calibration system. The updatedparameters can then again be read by steps 816-828, enabling the remoteuser to view the changes through the simulation environment. Thus, theexemplary method steps depicted in FIG. 8 permit a remote user tocalibrate a steerable interactive system in a target environment.

In view of the foregoing discussion, it will be appreciated that anadditional method step of obtaining the aforementioned precalibrationdata can be included. Furthermore, it will be appreciated that in block818, more generally, the parameters that are estimated could constitutespatial parameters associated with the camera and/or projector, and suchspatial parameters could be estimated based on the aforementionedprecalibration data and/or the redirection device parameters. Further,such spatial parameters could form at least part of the aforementionedimage and parameter data. The image and parameter data could alsoinclude, for example, camera and projector focus and zoom parameters,projector image data, and camera image data. The calibration data couldinclude image distortion data, and the updating step or operation couldinclude applying the image distortion data to the projector image data,and projecting an updated image based on the projector image data. Thecalibration data could also include focus data, zoom data, and pan andtilt data, and the updating step could also include driving aredirection device based at least in part on the pan and tilt data.Furthermore, an additional repeating step could be provided, wherein thestep of obtaining image and parameter data, the configuring step, thestep of obtaining calibration data, and the updating step are repeateduntil calibration of the interactive system associated with the targetenvironment is determined to be satisfactory.

As discussed above with respect to FIG. 4, in one specific exemplaryembodiment of the present invention, a single movable video projectorand camera may be employed. Attention should now be given to FIG. 9,which depicts an exemplary apparatus 900, according to one aspect of thepresent invention, for collocating a processing unit and a visual unit.Apparatus 900 could be employed to implement a number of differentsystems, including by way of example and not limitation, the apparatus400 depicted in FIG. 4. Apparatus 900 can include a processing unitreceiving portion 902 adapted to receive a processing unit, such as, forexample, a laptop computer. Apparatus 900 could also include a visualunit mounting portion 904 that is adapted to mount a visual unit suchas, by way of example and not limitation, a video projector, and/or avideo camera or the like. Apparatus 900 can further include a connectingbracket 906 having a first region 908 secured to the processing unitreceiving portion and a second region 910 secured to the visual unitmounting portion. The bracket 906 can define a channel adapted for flowof a cooling medium, and the channel can be in fluid communication withthe processing unit receiving portion 902, as will be discussed morefully below. The processing unit receiving portion 902 can be formedwith at least one aperture for passage of the cooling medium and thevisual unit mounting portion 904 and/or the second region of theconnecting bracket 910 can also be formed with one or more apertures forpassage of the cooling medium, for example, aperture 912.

The channel defined in the bracket can also be configured anddimensioned for passage of electrical cables in addition to (or, asdiscussed further below, instead of) the cooling medium. The visual unitmounting portion 904 can include a pan-tilt head portion 914 that issecured to the second region 910 of the connecting bracket 906, and canalso include a visual unit receiving portion 916 that is mounted to thepan-tilt head portion 914 for pan and tilt motion.

As shown in FIG. 9, the connecting bracket 906 can be formed as atubular frame. The tubular frame can have a substantially V-shapedportion with two branches 918, and each of the branches can have a leg920 depending therefrom and substantially orthogonal thereto. Bracket906 can further include a fixing plate that forms the second region 910of bracket 906. The fixing plate can be secured to the two branches 918.The two legs 920 can form the first region 908 of the bracket 906.

The aforementioned cooling medium can be, for example, ambient air. Thevisual unit mounting portion 904 and the second region 910 of theconnecting bracket 906 can be configured to be mounted substantiallyabove the processing unit receiving portion 902 and the first region 908of the connecting bracket such that heat generated by the processingunit within processing unit receiving portion 902 will cause the channelof the connecting bracket 906 to function as a chimney, inducing a flowof ambient air into the at least one aperture of the processing unitreceiving portion 902, through the channel 906, and out of the at leastone aperture 912 that is provided in the visual unit mounting portion904, or the second region 910 of the connecting bracket 906 (as shown inFIG. 9).

FIG. 10 shows apparatus 900 with the pan-tilt head portion 914 andvisual unit receiving portion 916 omitted for purposes for illustrativeclarity.

FIG. 11 shows apparatus 900 with the processing unit receiving portion902 in an open condition. Processing unit receiving portion 902 caninclude a cabinet portion 922 defining a processing unit receivingcavity that can receive a processing unit, such as laptop computer 924.The cavity can be in fluid communication with the channel in theconnecting bracket 906. It will be appreciated that in the bracket 906as described, there is one channel in each leg 920. A cover portion 926can also be included, and can be mounted for motion (such as rotarymotion) with respect to the cabinet portion 922, and configured anddimensioned to function as a work tray when in an open state, asdepicted in FIG. 11. If desired, the cabinet portion 922 and coverportion 926 can be configured and dimensioned so that the cover portion926 can be transitioned from the open state depicted in FIG. 11 to aclosed state depicted in FIG. 10, and secured in the closed statedepicted in FIG. 10, by a human operator without the use of tools, andpreferably without the use of fasteners.

The processing unit receiving portion 902 can be formed with sides 928,on the cabinet portion 922, for example. At least a portion of the twolegs 920 can extend along the sides 928 to enhance structural rigidityof the processing unit receiving portion. The processing unit receivingportion 902, the visual unit mounting portion 904, and the connectingbracket 908 can be formed with a predetermined number of cable accesspoints, as will be discussed below, and cable access can besubstantially limited to these cable access points, for example, toreduce the chance of damage and/or tampering.

The aforementioned operation without tools and/or fasteners can befacilitated by slots 930 formed in a projecting lip 932 on portion 922.Such projecting lip 932 and slots 930 may be formed at both upper andlower areas of portion 922 as shown in FIG. 11, for purposes to bediscussed below with respect to FIG. 12. Projections on lid 926 canengage with slots 930 via a snapping action such that engagement anddisengagement of lid 926 can be effectuated without the use of fixedhinges. The aforementioned opening 912 can function as both a cable portand an air vent. Opening 934 in portion 922 can function as an airinlet, permitting the interior channel in bracket 906 to act as achimney for ventilation purposes. Opening 936 can be provided forpurposes of a cable inlet, for example, for the power cable of thecomputer 924. Other cables, for example those for the projector, camera,and pan-tilt head can be internal to the enclosure 900 and can passthrough the interior of bracket 906. Perforated air vents 938 can alsobe provided for additional ventilation. Note that aperture 934 can beformed between the interior of portion 922 and the exterior.

As seen in FIG. 12, the cabinet portion 922, cover portion 926, andconnecting bracket 906 can be configured and dimensioned such that theycan be reformatted between a first format where the cabinet portion 922is below the fixing plate 910 as shown in FIG. 11, and a second formatshown in FIG. 12, where the cabinet portion 922 is above the fixingplate 910. The cover portion 926 can be hinged to the cabinet portion922 in a releasable manner, using the aforementioned slots andprojections, such that the cover portion 926 can pivot downward to forma work tray in either of the formats as shown in FIGS. 11 and 12. Itshould be noted that the orientation in FIG. 11 may be preferable forcooling purposes, as it may be preferable to have the heat sourcerepresented by computer 924 located at the bottom of the “chimney”formed by the bracket 906.

It should be noted that the channel in bracket 906 does not always needto be dimensioned to function as a channel for flow of a cooling medium.If desired, it could be formed as a cable raceway, and its functionalitycould be limited to that of a cable raceway. Alternatively, it couldfunction only as a “chimney.” However, it is believed preferable that itbe configured and dimensioned to serve both functions. It will befurther appreciated that the apparatus 900 is useful for otherapplications, for example, it could be employed to mount a plasmatelevision with an associated integrated notebook computer.

FIG. 13 shows the interior of cabinet portion 922 with a number ofpreviously-described elements. An outlet strip 940 and several powersupplies 942 can also be seen, and are typical of the flexible manner inwhich items can be located within portion 922.

Attention should now be given to FIGS. 14-16, which depict additionaldetails of the pan-tilt head portion 914. In one or more exemplaryembodiments of the present invention, the pan-tilt head portion 914 caninclude an upper ring portion 944 and a lower shell portion 946. Theupper ring portion 944 can typically be secured to the second region 910of the connecting bracket 906 using, for example, an appropriate bracketstructure 945, as best seen in FIG. 16. The lower shell portion 946 canbe mounted for panning motion with respect to the upper ring portion944. In one or more exemplary embodiments, the lower shell portion 946can be configured to remain stationary with respect to the upper ringportion 944 during tilt. Pan-tilt head portion 914 can also include apan motor unit 948 interposed between the upper ring portion 944 and thelower shell portion 946 to effect the panning motion. Further, pan-tilthead portion 914 can also include a tilt motor unit 950 secured to thelower shell portion 946 and adapted to mount the visual unit receivingportion for tilting motion with respect to the lower shell portion 946.

As best seen in FIG. 15, lower shell portion 946 can be formed with arecessed region 952 that is adapted to provide clearance for the visualunit receiving portion at relatively large tilt angles. The visual unitreceiving portion can be mounted, for example, at location 951, as bestseen in FIG. 15. As best seen in FIG. 16, an appropriate bracket 954 canbe included on the lower shell portion 946 and can be configured anddimensioned to receive and secure the tilt motor 950, preferably in amanner such that a human operator can secure the tilt motor 950 to thebracket 954 without the use of tools, and most preferably also withoutthe use of fasteners. Of course, designs employing tools and/orfasteners can also be used if desired. It will be appreciated that twobrackets 954 can be provided to embrace the tilt motor 950 such that thelower shell portion 946 pans along with the tilt unit. As noted,brackets 954 are preferably designed such that no screws or other partsare needed to fix the lower shell 946 to the tilt motor 950, thusfacilitating maintenance, assembly, and disassembly.

For purposes of illustrative convenience, only a single pan-tilt unitcable 947 is depicted in FIGS. 14 and 16. It will be appreciated thattypically numerous cables would be enclosed therein. Mounting bracket945 can be configured to serve two functions, namely, connecting theportions together by attaching to the pan motor 948, and reducing oreliminating vibration within the pan-tilt enclosure region. Note alsothat recessed region 952, in addition to allowing the full range oftilt, can be configured so as still to provide adequate room for cablerouting.

Attention should now be given to FIGS. 17 and 18, which depictadditional details of visual unit receiving portion 916. Portion 916 caninclude, for example, a projector enclosure 956 that can be mounted tolocation 951 of the pan-tilt head portion, for example, by mountingbracket 958. Visual unit receiving portion 916 can also include anoptical enclosure 960 that can be mounted to the projector enclosure956. As best seen in FIG. 18, a visual unit to be employed with one ormore exemplary embodiments of the present invention can include aprojector 962 having a projector lens 964, with an attached video camera966 having a camera lens 968. The projector enclosure 956 can beconfigured and dimensioned to receive the projector 962 with attachedvideo camera 966. The optical enclosure 960 can include, for example, alens housing 970 that can be secured to the projector enclosure 956, anda dust shield 972 that can be secured to the lens housing 970 in aposition so as to be substantially adjacent the projector lens 964 andthe camera lens 968 when the projector 962 and the camera 966 aremounted in the projector enclosure 956. Use of the dust shield 972, inaddition to providing protection for the lenses, enhances ease ofcleaning.

A bulb cover 974 can be provided to allow ready access to the bulb ofprojector 962. A rear cover/cable holder 976 can also be provided toenclose the rear portion of the projector enclosure 956, and to providerouting for cables and the like. The projector enclosure 956 can, ifdesired, be formed with a lower portion 978 and an upper cover portion980. Further, the lens housing 970 can be formed, if desired, from upperhalf 982 and lower half 984.

It will be appreciated that various configurations or assemblies fallwithin the scope of the present invention. For example, the elementsdescribed herein can be provided as an assembly or as a kit of parts.Furthermore, they can be provided together with a projector 962 and avideo camera 966, or without such items, which could be furnished by anend user or otherwise. Similarly, computer 924 can be included or couldbe provided otherwise. In one aspect, elements can be provided to form avisual apparatus for collocation with a processing unit. The visualapparatus could include the processing unit receiving portion 902 and avisual unit formed by projector 962 and camera 966 (as will be discussedmore fully in connection with FIG. 19 below, the camera 966 can bemounted so as to move with the projector 962). The visual apparatuscould also include a visual unit mounting portion 904 and the connectingbracket 906.

Attention should now be given to FIG. 19, which shows details of a tiltbracket and optical filter useful in one or more exemplary embodimentsof the present invention. Items similar to those discussed with regardto FIGS. 9-18 have received the same reference character, and will notbe separately discussed. As shown in FIG. 19, camera 966 can be mountedon a tilt bracket 986. The tilt of the camera 966 with respect to theprojector 962 may be important in order to optimize the viewing angle ofthe camera 966 with respect to the projector 962. The tilt bracket 986helps to ensure that the camera 966 is mounted at an appropriate(preferably optimal) viewing angle with respect to the projector 962 atthe time of assembly, and reduces or eliminates any need to adjust thecamera 966 during operation. Thus, camera 966 has a viewing angle withrespect to projector 962, and tilt bracket 986 interposed between camera966 and projector 962 is configured and dimensioned to attain apredetermined value of the viewing angle of the camera 966 with respectto the projector 962.

In one or more applications, interference between the optical systems ofthe projector 962 and the camera 966 may be significant. This mayespecially be so under varying lighting conditions. Such interferencemight result in artifacts in the image sensed by the camera 966 and, inturn, this might confound the ability to detect interactions whenprocessing the image. For example, in bright lighting conditions, theshutter speed of the camera 966 is normally automatically increased toreduce exposure time. A very short exposure time can result in thecamera sensing small changes in color of the projected image that occurover such short periods in, for example, DLP projectors such as theaforementioned XJ-450. Such sensing may cause the sensed image toartificially show sudden changes in color, which may in turn mayconfound the processing algorithm for detecting objects and theirinteractions through analysis of the sensed imaged as discussedhereinabove. In one or more exemplary embodiments of the presentinvention, an optical filter 988 can be provided for the lens 968 ofcamera 966. Such filter can reduce the intensity of light seen by thecamera 966, so as to reduce or eliminate interference between the opticsof the projector 962 and the camera 966. Such use of an optical filtermay be applicable to camera-projector systems in general, and mayenhance the effectiveness of such systems in sensing objects andinteractions.

As noted above, one or more elements described herein can be provided asan assembly or as a kit of parts. By way of example and not limitation,one such kit of parts could be provided for collocating a processingunit and a visual unit. The kit of parts could include a processing unitreceiving portion of the kind described above and a visual unit mountingportion of the kind described above. A connecting bracket of the kinddescribed above could also be provided and it could include a channelconfigured and dimensioned for passage of cables and/or flow of acooling medium. The parts can be designed such that, upon assembly, thechannel would be in communication with the processing unit receivingportion, and also with the visual unit mounting portion and/or thesecond region of the connecting bracket.

It will be appreciated that the processing unit receiving portion 902could be configured and dimensioned as desired to receive any type ofprocessing unit. In the exemplary embodiments depicted in the figures,the processing unit receiving portion 902 is formed to receive acomputer having dimensions and a form factor consistent with standardlaptop computers. It will also be appreciated that the exemplaryembodiment depicted in FIGS. 9-19 can provide effective mounting,modularity, portability and access for maintenance. The pan-tilt headportion can serve as the aforementioned redirection device for the videoprojector. Appropriate thermal control through ventilation and the“chimney” effect is also provided.

The details of the exemplary embodiment depicted in FIGS. 14-16 providebenefits for cable routing. Specifically, the design wherein the lowershell portion 946 can rotate independently of the upper ring 944 permitsmaximal pan and tilt coverage while minimizing twist of cables andresultant torque. Further, the exemplary components depicted in FIGS.9-19 may provide for seamless integration of one or more of a separatecomputer, power supply, power controller, cabling, and power stripcomponents in a single, aesthetically-pleasing unit that also integratesprojector and camera elements.

In view of the discussions herein, it will be appreciated that one ormore exemplary embodiments of the present invention can provide one ormore of the features and advantages discussed below. Interactions withthe environment can be detected by one or more devices that can bedispersed throughout the environment and need not be limited to a singlefixed camera tied to a projector. Further, a number of display devicescan be provided, and a plurality of display zones can be provided for aswell, including a physical display or displays or a moveable projectordisplay or displays, or any desired combination thereof. Interactions tobe sensed are not limited to interactions with a projected image;indeed, sensed interactions can be any place throughout the environmentand are not necessarily limited to the region of a projected (or other)image. An appropriate display or displays in an appropriate display zoneor zones can be activated or modified in response to actions sensedthroughout the environment. Such modification can include, but is notlimited to, position, orientation, content and type of display, and mayalso include steering and other activation activities. Coverage can beprovided, if desired, throughout the subject environment.

A variety of techniques, utilizing dedicated hardware, general purposeprocessors, firmware, software, or a combination of the foregoing may beemployed to implement the present invention. At present, it is believedthat the preferred implementation will make substantial use of softwarerunning on a general purpose computer. With reference to FIG. 20, suchan implementation might employ, for example, a processor 2002, a memory2004, and an input/output interface formed, for example, by a display2006 and a keyboard 2008. The term “processor” as used herein isintended to include any processing device, such as, for example, onethat includes a CPU (central processing unit) and/or other forms ofprocessing circuitry. Further, the term “processor” may refer to morethan one individual processor. The term “memory” is intended to includememory associated with a processor or CPU, such as, for example, RAM(random access memory), ROM (read only memory), a fixed memory device(e.g., hard drive), a removable memory device (e.g., diskette), a flashmemory and the like. In addition, the phrase “input/output interface” asused herein, is intended to include, for example, one or more mechanismsfor inputting data to the processing unit (e.g., mouse), and one or moremechanisms for providing results associated with the processing unit(e.g., printer). The processor 2002, memory 2004, and input/outputinterface such as display 2006 and keyboard 2008 can be interconnected,for example, via bus 2010 as part of a data processing unit 2012.Suitable interconnections, for example via bus 2010, can also beprovided to a network interface 2014, such as a network card, which canbe provided to interface with a computer network, and to a mediainterface 2016, such as a diskette or CD-ROM drive, which can beprovided to interface with media 2018.

Accordingly, computer software including instructions or code forperforming the methodologies of the invention, as described herein, maybe stored in one or more of the associated memory devices (e.g., ROM,fixed or removable memory) and, when ready to be utilized, loaded inpart or in whole (e.g., into RAM) and executed by a CPU. Such softwarecould include, but is not limited to, firmware, resident software,microcode, and the like.

Furthermore, the invention can take the form of a computer programproduct accessible from a computer-usable or computer-readable medium(e.g., media 2018) providing program code for use by or in connectionwith a computer or any instruction execution system. For the purposes ofthis description, a computer usable or computer readable medium can beany apparatus for use by or in connection with the instruction executionsystem, apparatus, or device.

The medium can be an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system (or apparatus or device) or apropagation medium. Examples of a computer-readable medium include asemiconductor or solid-state memory (e.g. memory 2004), magnetic tape, aremovable computer diskette (e.g. media 2018), a random access memory(RAM), a read-only memory (ROM), a rigid magnetic disk and an opticaldisk. Current examples of optical disks include compact disk-read onlymemory (CD-ROM), compact disk-read/write (CD-R/W) and DVD.

A data processing system suitable for storing and/or executing programcode will include at least one processor 2002 coupled directly orindirectly to memory elements 2004 through a system bus 2010. The memoryelements can include local memory employed during actual execution ofthe program code, bulk storage, and cache memories which providetemporary storage of at least some program code in order to reduce thenumber of times code must be retrieved from bulk storage duringexecution.

Input/output or I/O devices (including but not limited to keyboards2008, displays 2006, pointing devices, and the like) can be coupled tothe system either directly (such as via bus 2010) or through interveningI/O controllers (omitted for clarity).

Network adapters such as network interface 2014 may also be coupled tothe system to enable the data processing system to become coupled toother data processing systems or remote printers or storage devicesthrough intervening private or public networks. Modems, cable modem andEthernet cards are just a few of the currently available types ofnetwork adapters.

In any case, it should be understood that the components illustratedherein may be implemented in various forms of hardware, software, orcombinations thereof, e.g., application specific integrated circuit(s)(ASICS), functional circuitry, one or more appropriately programmedgeneral purpose digital computers with associated memory, and the like.Given the teachings of the invention provided herein, one of ordinaryskill in the related art will be able to contemplate otherimplementations of the components of the invention.

Although illustrative embodiments of the present invention have beendescribed herein with reference to the accompanying drawings, it is tobe understood that the invention is not limited to those preciseembodiments, and that various other changes and modifications may bemade by one skilled in the art without departing from the scope orspirit of the invention.

1. A computer-implemented method for remotely calibrating, from a remoteenvironment, an interactive system associated with a target environment,said method comprising the steps of: obtaining a three-dimensional modelof said target environment; obtaining image and parameter data of saidtarget environment; mapping said image and parameter data onto saidthree-dimensional model; and developing calibration data of said targetenvironment based on said mapping, said calibration data beingrepresentative of operation of said system, said operation includingsubject interactions with said system dispersed through said targetenvironment.
 2. The method of claim 1, wherein said developing stepcomprises: updating at least a portion of said image and parameter datato obtain updated image and parameter data; mapping said updated imageand parameter data onto said three-dimensional model; and configuringsaid updated image and parameter data for transmission to saidinteractive system associated with said target environment, as saidcalibration data.
 3. The method of claim 2, further comprising theadditional step of repeating said updating step, and said step ofmapping said updated image and parameter data, until said step ofmapping said updated image and parameter data indicates that saidupdated image and parameter data is satisfactory, wherein results ofsaid repeating of said steps are configured for said transmission. 4.The method of claim 1, wherein said image and parameter data comprisesfocus data, zoom data, pan and tilt data, image data from a camera insaid remote environment, and image data from a projector in said remoteenvironment.
 5. The method of claim 4, wherein said image and parameterdata further comprises displacement data.
 6. The method of claim 1,further comprising the additional step of building saidthree-dimensional model of said target environment.
 7. The method ofclaim 1, wherein said developing step comprises: displaying results ofsaid mapping step to a user; obtaining user input defining updates to atleast a portion of said image and parameter data, resulting in updatedimage and parameter data; repeating said displaying and obtaining userinput steps until said user determines that said updated image andparameter data is satisfactory; and configuring said updated image andparameter data for transmission to said interactive system associatedwith said target environment, as said calibration data.
 8. Acomputer-implemented method for remotely calibrating, from a remoteenvironment, an interactive system associated with a target environment,said method comprising the steps of: obtaining image and parameter dataof said target environment; configuring said image and parameter data ofsaid target environment for transmission to said remote environment;obtaining calibration data from said remote environment, saidcalibration data being representative of operation of said system, saidoperation including subject interactions with said system dispersedthrough said target environment; and updating display and sensoroperation of said interactive system based on said calibration data. 9.The method of claim 8, further comprising the additional step ofobtaining precalibration data.
 10. The method of claim 9, wherein saidstep of obtaining said image and parameter data comprises: readingredirection device parameters; and estimating camera and projectorspatial parameters based on said precalibration data and saidredirection device parameters; said camera and projector spatialparameters forming at least part of said image and parameter data. 11.The method of claim 10, wherein said step of obtaining said image andparameter data further comprises: reading camera and projector focus andzoom parameters; reading projector image data; and reading camera imagedata; said camera and projector focus and zoom parameters, saidprojector image data, and said camera image data forming at least partof said image and parameter data.
 12. The method of claim 8, whereinsaid calibration data comprises image distortion data, and wherein saidupdating step comprises: applying said image distortion data toprojector image data; and projecting an updated image based on saidprojector image data.
 13. The method of claim 12, wherein saidcalibration data further comprises focus data, zoom data, and pan andtilt data, and wherein said updating step further comprises driving aredirection device based on at least said pan and tilt data.
 14. Themethod of claim 8, further comprising the additional step of repeatingsaid obtaining image and parameter data step, said configuring step;said obtaining calibration data step; and said updating step untilcalibration of said interactive system associated with said targetenvironment is satisfactory.
 15. A computer program product comprising acomputer usable medium including computer usable program code forremotely calibrating, from a remote environment, an interactive systemassociated with a target environment, said computer program productincluding: computer usable program code for obtaining athree-dimensional model of said target environment; computer usableprogram code for obtaining image and parameter data of said targetenvironment; computer usable program code for mapping said image andparameter data onto said three-dimensional model; and computer usableprogram code for developing calibration data of said target environmentbased on said mapping, said calibration data being representative ofoperation of said system, said operation including subject interactionswith said system dispersed through said target environment.
 16. Thecomputer program product of claim 15, wherein said computer usableprogram code for developing includes: computer usable program code forupdating at least a portion of said image and parameter data to obtainupdated image and parameter data; computer usable program code formapping said updated image and parameter data onto saidthree-dimensional model; and computer usable program code forconfiguring said updated image and parameter data for transmission tosaid interactive system associated with said target environment, as saidcalibration data.
 17. The computer program product of claim 16, furthercomprising computer usable program code for repeating said updatingstep, and said step of mapping said updated image and parameter data,until said step of mapping said updated image and parameter dataindicates that said updated image and parameter data is satisfactory,wherein results of said repeating of said steps are configured for saidtransmission.
 18. The computer program product of claim 15, wherein saidcomputer usable program code for developing includes: computer usableprogram code for displaying results of said mapping step to a user;computer usable program code for obtaining user input defining updatesto at least a portion of said image and parameter data, resulting inupdated image and parameter data; computer usable program code forrepeating said displaying and obtaining user input steps until said userdetermines that said updated image and parameter data is satisfactory;and computer usable program code for configuring said updated image andparameter data for transmission to said interactive system associatedwith said target environment, as said calibration data.
 19. A computerprogram product comprising a computer usable medium including computerusable program code for remotely calibrating, from a remote environment,an interactive system associated with a target environment, saidcomputer program product including: computer usable program code forobtaining image and parameter data of said target environment; computerusable program code for configuring said image and parameter data ofsaid target environment for transmission to said remote environment;computer usable program code for obtaining calibration data from saidremote environment, said calibration data being representative ofoperation of said system, said operation including subject interactionswith said system dispersed through said target environment; and computerusable program code for updating display and sensor operation of saidinteractive system based on said calibration data.
 20. The computerprogram product of claim 19, further comprising computer usable programcode for repeating said obtaining image and parameter data step, saidconfiguring step; said obtaining calibration data step; and saidupdating step until calibration of said interactive system associatedwith said target environment is satisfactory.